DiLBERT: Cheap Embeddings for Disease Related Medical NLP

Electronic Health Records include health-related information, among which there is text mentioning health conditions and diagnoses. Usually, also coded using appropriate terminologies classifications. The act of coding time consuming prone to mistakes. Consequently, increasing demand for clinical mining tools help coding. In last few years Natural Language Processing (NLP) models has been shown be effective in sentence-level tasks. Taking advantage from the transfer learning capabilities those models, a number biomedicine specific have developed. However, biomedical can seen as too general some area like diagnostic expressions. this paper, we describe BERT model specialized on tasks related diagnoses conditions. To obtain disease-related language model, created pre-training corpora starting ICD-11 entities, enriched them with documents selected by querying PubMed Wikipedia entity names. Fine-tuning carried out towards three downstream two different datasets. Results show that our besides being trained much smaller than state-of-the-art algorithms, leads comparable or higher accuracy scores all considered tasks, particular 97.53% death certificate coding, 81.32% document are both slightly other models. summarize practical implications work, pre-trained fine-tuned domain small corpora, better performance This approach may simplify development languages English, due minor quantity data needed training.